Neal-Montgomery NLP System Evaluation Methodology

On what basis are the input processing capabilities of Natural Language software judged? That is, what are the capabilities to be described and measured, and what are the standards against which we measure them? Rome Laboratory is currently supporting an effort to develop a concise terminology for describing the linguistic processing capabilities of Natural Language Systems, and a uniform methodology for appropriately applying the terminology. This methodology is meant to produce quantitative, objective profiles of NL system capabilities without requiring system adaptation to a new test domain or text corpus. The effort proposes to develop a repeatable procedure that produces consistent results for independent evaluators.